The verification and validation (V&V) or testing phase of an engineering project is one of the major contributors to the overall development effort and cost of the project. Verification is the process of determining if a system meets the system requirements correctly. Validation is the process of evaluating the system to determine whether it satisfies the specified requirements and meets customer needs. Validation typically takes place after verification is completed. One part of verification and validation is generating test cases. A test case is a set of conditions or variables under which a test procedure determines if a requirement is satisfied. A number of test cases may be necessary to determine that the requirement is fully satisfied.
Past approaches to generating test cases have not been efficient, adding needless effort and cost to projects. In one approach, a verification engineer (a/k/a test engineer) specified tests based on requirements, which a systems engineer typically specified in text documents using natural language and simple graphics. The verification engineer manually specified the test cases by reading and interpreting the requirements, attempting to understand the coherence and interferences between the various requirements, and deriving test cases based on his gathered knowledge. Following the test case specification, the verification engineer wrote test scripts that automated the test cases.
Another approach to generating test cases used formal requirement specification instead of natural language. In this formal methods approach, formal models were generated from the formal requirement specifications and model checkers performed formal verification of the formal requirement specifications. Formal methods are mathematically based techniques (e.g., temporal logic formulas) for the specification, development and verification of software and hardware systems. Formal verification is the act of proving or disproving the correctness of a system with respect to a certain formal specification or property, using formal methods. Formal verification is becoming a key technology to ensure the quality of complex systems (e.g., hardware/software). Today, formal verification is widely used in the design phase of hardware projects. However, formal verification is more or less neglected in software and systems development. Model checking is a method for algorithmically verifying formal systems using a model often derived from a hardware or software design.
One kind of formal methods approach to generating test cases used a model checker called GateProp, which is available from OneSpin Solutions GmbH, Munich. In GateProp, most test cases started from a dedicated state, which was usually different from the initial state of the system (i.e., the “idle” state). The verification engineer specified a particular test case to test a specific aspect of the system that described this aspect, but did not describe how to reach the start state of the test case. This is problematic, because an automated test procedure needs to start from a well-defined idle state. To generate a test case starting from the idle state, it was first necessary to find out how to get from the idle state to the start state of the test case and, second, it was necessary to specify a very complex property for the model checker to generate the needed input sequences. This approach is time-consuming and error-prone. In addition, the generated input sequences contained a large number of unnecessary steps (often greater than 10 times what was required), which increased the runtime of the tests. Even if automated tests ran 24 hours a day, the test coverage achieved during the limited duration of the project was lower than it would have been without these unnecessary steps.
Another approach to generating test cases used model checkers to generate stimuli (inputs) and responses (outputs) for test cases. For example, formal models were generated by using a very high-speed integrated circuit hardware description language (VHDL) and checked by using a commercial VHDL model checker. Other approaches generated test cases from specifications without model checking but by simply determining all combinations or permutations of parameters, input/output signals, etc.
It is difficult—in most cases even impossible—to ensure that the generated test cases cover all requirements of a specific functionality regardless of the approach. For example, in telecommunications, the complexity of the implemented functions, protocols, and their interworking inherently causes this difficulty. In addition, the verification engineer may misinterpret natural language requirements, which themselves may be incomplete or ambiguous. This makes it even more difficult to derive the right test cases to provide sufficient test coverage. Furthermore, it is time consuming for the verification engineer to specify test cases and write test scripts. Test case generation using methods other than model checking typically are slow and create so large a number of test cases that they cannot be executed in a reasonable timeframe.
Accordingly, there is a need for more efficient generation of test cases to help reduce the effort and cost associated with V&V and to improve the time-to-market for project development.